Information processing method, information processing system, and recording medium

ABSTRACT

An information processing method performed by a computer includes: obtaining prediction results that are results of prediction performed by predictors on same input data; obtaining, for each of the prediction results, an influence that the input data had on the prediction result; determining, based on the prediction results, one or more combinations of the prediction results; and presenting, side by side or by superposition, using a presentation device, influences obtained for prediction results that are included in the prediction results and are in a same combination among the one or more combinations.

CROSS REFERENCE TO RELATED APPLICATIONS

This is a continuation application of PCT International Application No.PCT/JP2020/035522 filed on Sep. 18, 2020, designating the United Statesof America, which is based on and claims priority of Japanese PatentApplication No. 2019-218153 filed on Dec. 2, 2019. The entiredisclosures of the above-identified applications, including thespecifications, drawings and claims are incorporated herein by referencein their entirety.

FIELD

The present disclosure relates to an information processing method andthe like performed by a computer.

BACKGROUND

Techniques utilized to present detection results of objects appearing onan image are proposed (see, for example, PTL 1 and NPL 1).

CITATION LIST Patent Literature

-   PTL 1: Japanese Unexamined Patent Application Publication No.    2018-181273

Non Patent Literature

-   NPL 1: Erik Bochinski et al, “High-Speed tracking-by-detection    without using image information”, 14th IEEE International Conference    on Advanced Video and Signal Based Surveillance (AVSS), August 2017

SUMMARY Technical Problem

According to the conventionally proposed techniques, it is difficult toconcurrently evaluate respective behaviors of different predictors thatrespectively perform prediction processes such as an object detectionprocess.

The present disclosure provides an information processing method and thelike capable of concurrently evaluating respective behaviors ofdifferent predictors.

Solution to Problem

An information processing method according to an aspect of the presentdisclosure is a method performed by a computer and includes: obtainingprediction results that are results of prediction performed bypredictors on same input data; obtaining, for each of the predictionresults, an influence that the input data had on the prediction result;determining, based on the prediction results, one or more combinationsof the prediction results; and presenting, side by side or bysuperposition, using a presentation device, influences obtained forprediction results that are included in the prediction results and arein a same combination among the one or more combinations.

Also, an information processing system according to an aspect of thepresent disclosure includes: a prediction result obtainer that obtainsprediction results that are results of prediction performed bypredictors on same input data; an input data influence obtainer thatobtains, for each of the prediction results, an influence that the inputdata had on the prediction result; a determiner that determines, basedon the prediction results, one or more combinations of the predictionresults; and an influence presenter that presents, side by side or bysuperposition, using a presentation device, influences obtained forprediction results that are included in the prediction results and arein a same combination among the one or more combinations.

Note that these general or specific aspects may be implemented using,other than the method and system described above, a device, anintegrated circuit, a computer-readable recording medium such as CD-ROM,or any combination of devices, systems, integrated circuits, methods,computer programs, and recording media.

Advantageous Effects

An information processing method and the like according to the presentdisclosure make it possible to concurrently evaluate respectivebehaviors of different predictors.

BRIEF DESCRIPTION OF DRAWINGS

These and other advantages and features will become apparent from thefollowing description thereof taken in conjunction with the accompanyingDrawings, by way of non-limiting examples of embodiments disclosedherein.

FIG. 1 is a schematic diagram of a screen to which an informationprocessing method according to an embodiment can be applied.

FIG. 2 is a schematic diagram for describing a difference in objectdetection results between different object detection models.

FIG. 3A is a diagram schematically illustrating object detection resultsfor a common input image by three object detection models.

FIG. 3B is a diagram illustrating an example of data of an objectdetection result set outputted by one of the object detection models.

FIG. 4A is a diagram schematically illustrating an example of the stateafter the information processing method according to the embodiment isapplied to the object detection results illustrated in FIG. 3A.

FIG. 4B illustrates an example of the data of the object detectionresult set after the information processing method according to theembodiment is applied.

FIG. 5A is a diagram for describing a result of a process of insertingsubstitution data in the information processing method according to theembodiment.

FIG. 5B is a diagram illustrating an example of the data of the objectdetection result set corresponding to the result of the processillustrated in FIG. 5A.

FIG. 6 is a diagram for describing an aspect of the process of insertingthe substitution data in the information processing method according tothe embodiment, which is different from the example illustrated in FIG.5A.

FIG. 7A illustrates a display example on a UI screen to which theinformation processing method according to the embodiment is applied.

FIG. 7B illustrates a display example on a UI screen to which theinformation processing method according to the embodiment is applied.

FIG. 7C illustrates a display example on a UI screen to which theinformation processing method according to the embodiment is applied.

FIG. 7D illustrates a display example on a UI screen to which theinformation processing method according to the embodiment is applied.

FIG. 8 is a block diagram illustrating a configuration example of acomputer that performs the information processing method according tothe embodiment.

FIG. 9A is a schematic diagram for describing an analysis frame obtainedin the above-mentioned information processing method.

FIG. 9B illustrates an example of the data of the object detectionresult set in which data concerning the above-mentioned analysis frameis added.

FIG. 10 is a flow chart illustrating procedures of the informationprocessing method according to the present embodiment.

DESCRIPTION OF EMBODIMENT (Underlying Knowledge Forming Basis of thePresent Disclosure)

The present inventors found out that the above-mentioned conventionallyproposed techniques had the following problems.

For example, the number of subjects that are detected (hereinafter, alsoreferred to as detected subjects) may be different among objectdetections that are respectively executed on the same image byprediction models. In such a case, if a creator of the prediction modelstries to compare and analyze detection results for each detected subjectamong the prediction models, it is troublesome only to find out resultsfor the same detected subject from the detection results outputted bythe prediction models, and this is inconvenient and inefficient.Moreover, even if the number of the detected subjects is the same,because an estimated position or an appearing range of a given detectedsubject may be different among the prediction models, association workby visual checking is troublesome and is likely to cause a mistake, andthis is inefficient.

An information processing method according to an aspect of the presentdisclosure, which has been conceived in view of such inefficientsituations, is a method performed by a computer and includes: obtainingprediction results that are results of prediction performed bypredictors on same input data; obtaining, for each of the predictionresults, an influence that the input data had on the prediction result;determining, based on the prediction results, one or more combinationsof the prediction results; and presenting, side by side or bysuperposition, using a presentation device, influences obtained forprediction results that are included in the prediction results and arein a same combination among the one or more combinations.

According to this, for example, prediction results for a common targetare selected and combined from among the respective prediction resultsby the prediction models (predictors). Then, the influences that theinput data had on the prediction results included in this combinationare presented at a time, side by side or by superposition. Consequently,the respective behaviors (that is, influences) of the differentpredictors can be concurrently evaluated. As a result, a user's troubleof analyzing the prediction results is reduced compared with theconventional case. Accordingly, the user can perform the analysis of theprediction results more efficiently compared with the conventional case.

Furthermore, each of the predictors may be an object detector, each ofthe prediction results may be an object detection result set thatincludes object detection results, the influence obtained for each ofthe prediction results may be an influence that the input data had oneach of the object detection results included in the object detectionresult set, each of the one or more combinations may include objectdetection results included in different object detection result setsamong the object detection result sets, and the influences presentedside by side or by superposition may be influences obtained for theobject detection results included in the same combination.

According to this, the influences that the input data had on the objectdetection results included in this combination are displayed at a time,side by side or by superposition. As a result, a user's trouble beforereaching an analysis of the object detection results is reduced comparedwith the conventional case, and hence the user can perform the analysisof the object detection results more efficiently compared with theconventional case. This is particularly meaningful because objects aredetected by each object detector during the object detection and thetrouble of analyzing thus becomes enormous.

Furthermore, each of the object detection results may include a classthat is based on an object detected, and in each combination among theone or more combinations, the class included in each of the objectdetection results included in the combination may be a class that iscommon to other object detection results included in the combination.

According to this, in the case where the object detection is executedfor a plurality of classes, for example, even if estimated positions ofdetected subjects are close to each other, if the classes are different,presenting influences to the user at a time can be avoided. As a result,the user's trouble before reaching the analysis of the object detectionresults is reduced compared with the conventional case, and hence theuser can perform the analysis of the object detection results moreefficiently compared with the conventional case.

Here, for example, each of the object detection results may include adetection frame, and the one or more combinations may be determinedbased on overlap of or a positional relationship between detectionframes included in the object detection results included in differentobject detection result sets. Furthermore, each of the object detectionresults may include a detection likelihood, and the one or morecombinations may be determined further based on a degree of similaritybetween detection likelihoods included in the object detection resultsincluded in different object detection result sets.

According to this, a combination of detection results for a commondetected subject can be established more reliably from among therespective object detection results by the object detectors.

Furthermore, the presenting of the influences may include presenting,side by side, substitution data and the influences obtained for theobject detection results included in the same combination, when the samecombination does not include the object detection results included in anobject detection result set of an object detector among the objectdetectors. In addition, the information processing method according toan aspect of the present disclosure may further include presenting, sideby side, substitution data and an influence obtained for an isolatedobject detection result when the object detection results include theisolated object detection result, the isolated object detection resultbeing an object detection result not included in any of the one or morecombinations.

According to this, in the case where the number of the detected subjectsis different among the object detectors, the user can easily understand,for a given detected subject, which object detector did not detect thegiven detected subject or which object detector is the only objectdetector that detected the given detected subject.

Furthermore, the presenting of the influences may include presenting: aninfluence obtained for a reference object detection result that is oneof the object detection results included in a reference object detectionresult set that is one of the object detection result sets; andinfluences obtained for the object detection results included in acombination among the one or more combinations that includes thereference object detection result. In addition, the informationprocessing method according to an aspect of the present disclosure mayfurther include: receiving an operation of selecting the referenceobject detection result set; and switching the reference objectdetection result set to the object detection result set selected by theoperation.

According to this, the user can perform the analysis of the objectdetection results more efficiently compared with the conventional case,while focusing on a specific object detector. Moreover, the user canperform the analysis more efficiently compared with the conventionalcase, while changing the object detector to be focused on.

In addition, the information processing method according to an aspect ofthe present disclosure may further include: receiving an operation ofselecting the input data; and switching the input data to the input dataselected by the operation, wherein the presenting of the influences mayinclude presenting influences obtained for the prediction results ofprediction performed by the predictors on the input data selected.

According to this, even in the case where the number of pieces of theinput data is more than one, the user can perform the analysis of therespective object detection results by the object detectors moreefficiently compared with the conventional case.

In addition, the information processing method according to an aspect ofthe present disclosure may further include: receiving an operation ofselecting a group of the object detection results; and switching theobject detection results corresponding to the influences presented orthe object detection results presented, to object detection results ofthe group selected by the operation.

According to this, the user can select the object detection resultsrelating to presentation on a UI screen, depending on attributes, andthus can perform the analysis of the object detection results moreefficiently compared with the conventional case.

Furthermore, the presenting of the influences may further includepresenting, for each of the influences presented, information indicatinga predictor among the predictors that has output the prediction resultcorresponding to the influence.

According to this, the user can discriminate to which predictor amongthe predictors the influences presented relate.

Furthermore, an information processing method according to an aspect ofthe present disclosure is a method performed by a computer, and mayinclude: obtaining object detection result sets each including objectdetection results that are results of detection performed by objectdetectors on same input data; determining, based on the object detectionresults included in the object detection result sets, one or morecombinations of the object detection results included in the objectdetection result sets, the one or more combinations each including oneobject detection result of each of the object detection result sets; andpresenting, side by side or by superposition, using a presentationdevice, object detection results that are included in the objectdetection results in the object detection result sets and are in a samecombination among the one or more combinations.

According to this, the respective behaviors (that is, object detectionresults) of the different object detectors can be concurrentlyevaluated. As a result, the user can perform the comparison among therespective object detection results by the object detectors moreefficiently compared with the conventional case.

Furthermore, an information processing system according to an aspect ofthe present disclosure includes: a prediction result obtainer thatobtains prediction results that are results of prediction performed bypredictors on same input data; an input data influence obtainer thatobtains, for each of the prediction results, an influence that the inputdata had on the prediction result; a determiner that determines, basedon the prediction results, one or more combinations of the predictionresults; and an influence presenter that presents, side by side or bysuperposition, using a presentation device, influences obtained forprediction results that are included in the prediction results and arein a same combination among the one or more combinations.

According to this, for example, prediction results for a common targetare selected and combined from among the respective prediction resultsby the prediction models (predictors). Then, the influences that theinput data had on the prediction results included in this combinationare presented at a time, side by side or by superposition. Consequently,the respective behaviors of the different predictors can be concurrentlyevaluated. As a result, a user's trouble of analyzing the predictionresults is reduced compared with the conventional case. Accordingly, theuser can perform the analysis of the prediction results more efficientlycompared with the conventional case.

Note that these general or specific aspects may be implemented using,other than the methods and system described above, a device, anintegrated circuit, a computer-readable recording medium such as CD-ROM,or any combination of devices, systems, integrated circuits, methods,computer programs, and recording media.

Hereinafter, an embodiment of an information processing method and aninformation processing system according to an aspect of the presentdisclosure will be described with reference to the accompanyingdrawings. The embodiment described herein illustrates a specific exampleof the present disclosure. Therefore, the numerical values, shapes,constituent elements, the arrangement and connection of the constituentelements, steps (processes), the processing order of the steps, etc.illustrated in the embodiment below are mere examples, and do not intendto limit the present disclosure. In addition, among the constituentelements in the embodiment below, constituent elements not recited inany one of the independent claims are constituent elements which can beoptionally added. Moreover, the drawings are schematic diagrams and arenot necessarily precise illustrations.

EMBODIMENT

FIG. 1 is a schematic diagram of a screen to which an informationprocessing method according to an embodiment can be applied. This screenfunctions as a user interface (hereinafter, represented as a UI) ofapplication software used to analyze results of prediction bypredictors. This application software may be a web application thatdisplays this screen (hereinafter, referred to as a UI screen) on a webbrowser, and may be a native application or a hybrid application. Thisapplication software is executed by, for example, a processor includedin a computer for business or personal purposes of various types such asa tower type, a desktop type, or a tablet type, and displays the UIscreen on a monitor used by the user.

UI screen 10 receives a user's operation and displays the results ofprediction by the predictors or information pertaining to the results inaccordance with this user's operation. Note that the present embodimentis described by taking, as an example, the case where the predictors areobject detectors that detect objects appearing on input images indicatedby input data. The information pertaining to the results of predictionin this example is, for example: information concerning the existence ornon-existence of an influence that each portion of the input data had oneach of the results of prediction by the object detectors; andinformation concerning any or both of the size and the direction(whether the influence is positive or negative) of the influence ifthere is any influence. In other words, the influence is, for example, aresponse that is made by each predictor on the input data in aprediction process for outputting each prediction result. Alternatively,the influence may be expressed as an analysis result (that is, ananalysis frame or analysis value) of the response.

UI screen 10 is divided into three portions of an input data field, amodel field, and a result field in the stated order from the left. Theinput data field includes input data selector 20A. The model fieldincludes model selector 20B and display result selector 20C. The resultfield includes result display 40, display data switcher 50A, analysisframe collective switcher 50B, and analysis frame individual switcher50C.

Input data selector 20A presents candidates of the input data to bedisplayed in the result field with the prediction results or theanalysis results (which are hereinafter also referred to as resultinformation without being discriminated from each other) beingsuperposed on the input data, and allows the user to select any of thecandidates. The input data in this example is data of images, and theinput data is presented as thumbnails of the images. Thumbnails ofimages selected by the user (all the images in the example illustratedin FIG. 1) are each surrounded by a thick frame. Hereinafter, the inputdata is also referred to as an input image.

Model selector 20B presents candidates of the predictors whose resultinformation is to be displayed, and allows the user to select any of thecandidates. The predictors here mean prediction models (hereinafter,also referred to simply as models, for the sake of convenience) ofmachine learning. In the example illustrated in FIG. 1, the user narrowsthe candidates of the prediction models down to candidates for thepurpose of object detection, and then selects two models of “Model A”and “Model E”. Note that, even if the models selected by model selector20B in this way are common in the purpose, the selected models aredifferent in: the category depending on a difference in a trainingmethod or a network configuration; a data set used for training; or theamount of training.

Display result selector 20C allows the user to select items of theresult information to be displayed. In the case of the object detection,what the user is allowed to select as the items of the resultinformation is, for example, the category (class) of a detected object.In the example illustrated in FIG. 1, a car, a bicycle, a pedestrian,and a motorcycle are presented as selectable classes, and the pedestrianis selected. Moreover, in the example illustrated in FIG. 1, what theuser is allowed to also select as another item of the result informationto be displayed is a result type (TP: True Positive, FP: False Positive,and FN: False Negative). In the example illustrated in FIG. 1, TP and FNare selected.

After such selections in input data selector 20A, model selector 20B,and display result selector 20C, if the user clicks or taps a displaybutton placed below display result selector 20C, the result informationis displayed in result display 40 in the result field, depending on thecontents of the selections. In the example illustrated in FIG. 1, one ofthe input images selected in input data selector 20A and informationconcerning an object detection result corresponding to the predictionresult for this input image are displayed in result display 40 togetherwith the name of each of the models for object detection that outputsthe object detection result.

In the example illustrated in FIG. 1, two display images are displayedin result display 40 while being arranged in the left-right direction,and influences on the prediction results for one of the input imagesselected in input data selector 20A, by the two models (Model A andModel E) selected in model selector 20B are respectively superposed onthe two display images. Note that the display images may be displayedwhile being arranged in the top-bottom direction.

Which of the input images selected in input data selector 20A is to bedisplayed in result display 40 can be switched by using display dataswitcher 50A. In the example illustrated in FIG. 1, display dataswitcher 50A is a slider, and the input image displayed in resultdisplay 40 is switched depending on the position of a knob moved by theuser. Along with the switching of the input images, the resultinformation superposed on the input image is also switched. Moreover, inthis example, the name of the input image currently displayed in resultdisplay 40 and the order thereof among the selected input images aredisplayed above display data switcher 50A.

Analysis frame collective switcher 50B and analysis frame individualswitcher 50C allow the user to perform switching of analysis framescorresponding to an aspect of the influences on the prediction results,the analysis frames being respectively superposed on the input imagesand displayed as part of the display images in result display 40. Theswitched analysis frame is superposed on each input image, whereby adisplay image is generated. For example, if each model executes objectdetection on one input image and multiple objects are detected, oneanalysis frame is generated for each of the detection results (that is,detection frames). Note that what is superposed at a time on each inputimage in result display 40 is one of the analysis frames generated forthe detection frames. In the example illustrated in FIG. 1, if the useronce clicks or taps (hereinafter, also expressed as “presses” withoutdiscriminating between a click operation and a tap operation) any of anupward triangle and a downward triangle that are components of analysisframe collective switcher 50B provided only one to the result field, theanalysis frames respectively superposed on the input images displayed inresult display 40, among the analysis frames generated for therespective detection results by the models, are collectively switchedone after another. Moreover, analysis frame individual switcher 50C thatis the slider is provided for each input image (that is, each displayimage) on which the analysis frame concerning each model is superposed,in result display 40. If the user moves the knob of the slider, theanalysis frame superposed on the input image can be individuallyswitched for each image.

With the application of the information processing method according tothe present embodiment to this UI screen 10, an analysis of predictionresults can be more efficiently performed using UI screen 10. In anotherway of saying, the user can more efficiently perform the analysis of theprediction results using the UI screen that is operable in a mannersimilar to the conventional case.

There is often such a difference as described above among the predictionresults for the same input data, by the models. For example, in objectdetection, the number, estimated positions in each image, or appearingranges of subjects that are detected (hereinafter, also referred to asdetected subjects) can be different among the models. FIG. 2 is aschematic diagram for describing such a difference in object detectionresults between the models. In FIG. 2, illustrated are detection frameseach surrounding a pedestrian or a car that is a detected subject, amongrespective object detection results for input image Image 1 by the twomodels of Model A and Model B. Comparing the object detection results bythe two models, the number of the detected pedestrians is different, andthe sizes and ranges of detection frames that seem to be a detectionframe of a common pedestrian are similar but are not coincident. Notethat the third detection frame from the top by Model B surrounds atraffic cone on a road that is erroneously detected as a pedestrian or acar, and the third detection frame is an example of result type FP.Then, a car appearing in the right end of each image is detected byModel A but is not detected by Model B. Moreover, these detection framesare arranged from the top in order in which these detection frames areincluded in data of the detection results outputted by each model, andthe arrangement order may be different between the models as illustratedin FIG. 2.

The user uses UI screen 10 to compare and analyze, for each detectedsubject, the object detection results outputted by the models. At thistime, a request to arrange and display results for a common detectedsubject at a time on UI screen 10 can be made. Conventionally, in such acase, the user uses analysis frame individual switcher 50C to find outdetection results that can be regarded as detection results for thecommon detected subject (for example, a pedestrian appearing near theleft end of each image), from the object detection results outputted byeach model, and the user causes the found detection results to bedisplayed in result display 40. Alternatively, the user rearranges andadjusts the object detection results outputted by each model in his/herdesired order, and then uses analysis frame collective switcher 50B,although an operator therefor is not illustrated on UI screen 10.

The information processing method according to the present embodimentserves to save a user's trouble in this rearrangement in a sense.Hereinafter, the rearrangement according to this method is described indetail using examples.

FIG. 3A is a diagram schematically illustrating object detection resultsfor a common input image by three object detection models. Solid linerectangles each represent the entire region of the input image that is atarget of an object detection process. Moreover, a broken line frame oran alternate long and short dash line frame in each solid line rectangleis a detection frame obtained as a result of the object detectionprocess by each model, and different types of lines represent detectionframes of detected subjects in different classes, for example, apedestrian and a car. The arrangement in the top-bottom direction ofthese detection frames corresponds to the order in which pieces of dataconcerning the detection frames are arranged in the data of the objectdetection results outputted by each object detection model. FIG. 3Billustrates an example of the data of the object detection resultsoutputted by Model A that is one of the models. Each row of from thesecond row to the sixth row of this data includes an output valueconcerning each detection frame. The content of this output value showsthe position and size (range) in the entire image region and the classof the detected subject, of each of the detection frames included in theobject detection results by Model A in FIG. 3A. Similar data is alsooutputted by Models B and C that are other object detection models. Notethat, hereinafter, the object detection results outputted by each of theobject detection models are also referred to collectively as an objectdetection result set. That is, the data illustrated in FIG. 3B is thedata of the object detection result set formed from five detectionframes obtained as a result of executing object detection on one inputimage by Model A. Note that the illustration in FIG. 3B is an example ofthe structure and numerical values of such data, and the form andnumerical values of the data of the object detection result set are notlimited to this example. Moreover, items included in each objectdetection result are not limited to this example. For example, theabove-mentioned result type may be included in the object detectionresult. The same applies to FIG. 4B, FIG. 5B, and FIG. 9B to be used inthe following description.

If this input image is selected as a display target in input dataselector 20A on UI screen 10, the object detection results (detectionframes) included in the display image are switched in result display 40in the arrangement order thereof in the top-bottom direction of thefigure, through operations using analysis frame collective switcher 50Band analysis frame individual switcher 50C.

FIG. 4A is a diagram schematically illustrating an example of the stateafter the information processing method according to the presentembodiment is applied to the object detection results illustrated inFIG. 3A. In the example illustrated in FIG. 4A, the detection framescorresponding to the object detection results by each object detectionmodel are sorted depending on the classes of the detected subjects. Thisreflects that, according to this information processing method, theoutput values of each of the detection frames are rearranged dependingon the classes (“car” and “pedestrian”) of the detected subjects, in thedata of the object detection result set outputted by each objectdetection model. For example, as illustrated in FIG. 4B, the outputvalues of the detection frames in from the second row to the sixth roware rearranged in the order of the classes (“car” and “pedestrian”) ofthe detected subjects, in the data of the object detection result setoutputted by Model A illustrated in FIG. 3B. Similar rearrangement isperformed also in the data of the object detection result sets outputtedby Models B and C.

FIG. 5A is a diagram for describing a result of a process that isperformed subsequently to the above-mentioned sorting performeddepending on the classes of the detected subjects, in the informationprocessing method according to the present embodiment. The resultillustrated in FIG. 5A is described while focusing on different pointsfrom the state illustrated in FIG. 4A.

With reference to FIG. 5A, detection results for a common detectedsubject (or detection results that are highly likely to be the detectionresults for the common detected subject), among the object detectionresult sets respectively outputted by the object detection models ofModels A, B, and C, are placed at the same position (order) in thetop-bottom direction.

In order to arrange the object detection results in such a manner asdescribed above, the arrangement order of the object detection resultsin the object detection result set outputted by any of the models isdefined as a reference, and the object detection results in the objectdetection result sets respectively outputted by the other models arerearranged based on the defined reference. At this time, based onoverlap of detection frames in the same class among the detection framesincluded in the object detection result sets respectively outputted bytwo models, the arrangement order of the object detection results in thedetection result set outputted by the model that is not defined as thereference of the arrangement order may be determined. The overlap of thedetection frames means, for example, Intersection over Union (IoU), thatis, the ratio of the intersection to the union of the detection frames.Alternatively, more simply, based on the size of an overlapping portionof detection frames in the same class, whether or not to place theobject detection results in the same order may be determined.

Moreover, as another example of the method of determining thearrangement order of the object detection results in the objectdetection result set, the arrangement order thereof may be determinedbased on a positional relationship between detection frames in the sameclass among the object detection result sets respectively outputted bythe object detection models. As a specific example based on thepositional relationship between the detection frames, the distancebetween corresponding vertexes or corresponding sides of two detectionframes or the distance between geometric centers of the two detectionframes may be used.

Note that placing the detection results in the same order for eachdetected subject in the object detection result sets respectivelyoutputted by the object detection models as described above is alsoreferred to as determining a combination of the object detection results(or prediction results), in the present disclosure. In another way ofsaying, two or three object detection results included in the same rowin FIG. 5A are included in the same combination. The object detectionresults included in the same combination are displayed at a time inresult display 40 without a user's trouble of finding and rearranging.In the example illustrated in FIG. 5A, the detection frames that areplaced in the same row and correspond to the object detection results bythe respective models are detection frames in the same class, and thesedetection frames are larger in the IoU or closer in the position in eachimage than the detection frames placed in other rows.

Whether or not the detection results by the different models are to beincluded in the same combination based on the overlap of the detectionframes such as the IoU or based on the positional relationship betweenthe detection frames may be determined by, for example, comparing theoverlap or the distance with a predetermined threshold value. Even ifgiven detection results are detection frames in the same class and arelarger in the IoU than the detection frames placed in other rows, in thecase where the IoU falls below the predetermined threshold value, it isdetermined that the given detection results are not to be included inthe same combination. As another example, it may be determined byachieving overall optimization according to the Hungarian algorithm orthe like.

Note that a method of selecting the object detection result set(hereinafter, also referred to as a reference object detection resultset) that is used as the reference of the rearrangement of the objectdetection results in each object detection result set, by a computerthat performs the information processing method according to the presentembodiment is not particularly limited. For example, the user may beallowed to select the reference object detection result set.Alternatively, for example, an object detection result set in which thenumber of the detection results (detection frames) is the smallest or anobject detection result set in which the number thereof is the largestmay be selected as the reference object detection result set.Alternatively, for example, it may be randomly selected at the time ofthe rearrangement. FIG. 5A illustrates the result obtained by using, asthe reference, the order of the object detection results in the objectdetection result set by Model A in such a process performed in theinformation processing method according to the present embodiment.

Moreover, the result illustrated in FIG. 5A includes dotted rectanglesthat are not in FIG. 4A. In the case where there is a detected subjectthat has been detected by only part of the models selected in modelselector 20B, these rectangles correspond to data (hereinafter, referredto as substitution data) indicating substitutive information that isinserted in the object detection result sets by the models that have notdetected this detected subject. FIG. 5B is a diagram illustrating anexample of inserting the substitutive information in the objectdetection result set outputted by Model A. The “{blank}” in each of thefourth row and the fifth row in FIG. 5B is an example of the insertedsubstitutive information. This information corresponds to the two dottedrectangles existing in the column of the object detection results byModel A in FIG. 5A. The substitution data is presented side by side withthe display images based on the object detection results by the othermodels in result display 40, whereby information that this detectedsubject has not been detected by this model is presented to the user.Such substitution data may be presented as, for example, a blank frameor a simple frameless blank region, and may be presented as, forexample, characters, symbols, patterns, or figures indicating thenon-detection, in result display 40.

Moreover, FIG. 6 is a diagram for describing an aspect of inserting thesubstitution data, which is different from the example of FIG. 5A. Alsoin this example, what is used as the reference of the arrangement orderof the object detection results is the arrangement order of the objectdetection results in the object detection result set by Model A.However, in FIG. 6, the substitution data is not inserted in the objectdetection result set by Model A, and the state illustrated in FIG. 4Aafter the sorting based on the classes is maintained. The detectionresults for the detected subjects that have not been detected by ModelA, among the object detection results included in the object detectionresult sets by Model B and Model C are gathered at the end (lowermostportion) of the arrangement order.

Such rearrangement (sorting) of the object detection results in theobject detection result set may be automatically executed by, forexample, pressing a display button by the user. Alternatively, UI screen10 may be provided with an operation component (not illustrated) forreceiving an instruction to execute such rearrangement from the user,and the rearrangement may be executed in response to an operation onthis operation component.

Note that the fourth one and the sixth one from the top by Model A inFIG. 5A and the fourth one from the top by Model A in FIG. 6 correspondto the detection results (hereinafter, referred to as isolated objectdetection results) for the detected subjects that have been detected byonly any one of the three models. The isolated object detection resultsare not included in any of the combinations of the object detectionresults determined as described above. Display images based on suchisolated object detection results may also be presented side by sidewith the substitution data in result display 40.

A display example on UI screen 10 after such rearrangement is performedis illustrated in FIG. 7A to FIG. 7D. Part of result display 40 andanalysis frame collective switcher 50B in the result field of UI screen10 are extracted and illustrated in FIG. 7A to FIG. 7D. For the targetimage of the object detection, see image Image 1 in FIG. 2. In thisexample, the case where the three models of Model A, Model B, and ModelC execute the object detection on image Image 1 is assumed.

With reference to FIG. 7A, in this state, presented side by side aredisplay images each including an analysis result for a detection resultof a pedestrian that has been detected by all of the three models andappears a little closer to the left end from the center. In the exampleof the present disclosure, the analysis result (the size and thedirection of an influence) that is called the analysis frame in theabove is schematically represented by a patterned region superposed oneach image. A portion on which no analysis frame is superposedcorresponds to the fact that this portion of the input image has noinfluence on the object detection result or has an extremely smallinfluence thereon.

FIG. 7B illustrates the state of result display 40 after the downwardtriangle included in analysis frame collective switcher 50B is pressedonce in the state illustrated in FIG. 7A. Presented are display imageseach including an analysis result for a detection result of a pedestrianthat is included in both of the object detection result setsrespectively outputted by Model A and Model B and appears close to theleft end of the input image. Moreover, for Model C, this pedestrian hasnot been detected, and hence a frame including a figure as substitutiondata is displayed side by side with the other display images.

FIG. 7C illustrates the state of result display 40 after the downwardtriangle included in analysis frame collective switcher 50B is pressedfurther once in the state illustrated in FIG. 7B. In this state,presented side by side are: a display image including an analysis resultfor a detection result of a pedestrian included only in the objectdetection result set outputted by Model B; and frames each including afigure as substitution data, for Model A and Model C.

FIG. 7D illustrates the state of result display 40 after the downwardtriangle included in analysis frame collective switcher 50B is pressedfurther once in the state illustrated in FIG. 7C. In this state,presented side by side are: two display images each including ananalysis result for a detection result of a car commonly included in theobject detection result set by Model A and the object detection resultset by Model C; and a frame including a figure as substitution data, forModel B.

In this way, in the information processing method according to thepresent embodiment, the combinations of the prediction results by themodels are determined based on the overlap of or the positionalrelationship between the detection frames corresponding to the objectdetection results included in the object detection result sets, in theabove-mentioned example. The influences that the input image had on theobject detection results included in the same combination among thecombinations thus determined, in the object detection process by eachmodel are presented side by side to the user. According to this feature,without troubles of finding detection results for the same detectedsubject and rearranging the found detection results, the user can lookat and analyze the detection results for the same detected subject atone time.

Next, description is given of a configuration of a computer thatexecutes such rearrangement and the information processing methodaccording to the present embodiment that is performed for thisrearrangement by the computer. FIG. 8 is a block diagram illustrating aconfiguration example of computer 100 that performs the informationprocessing method according to the present embodiment.

Computer 100 is the above-mentioned computer of various types, andincludes information processing device 80 and display device 60.

Information processing device 80 is configured by: a storage in whichthe above-mentioned application software is stored; and an arithmeticprocessor that reads out and executes this application software.Information processing device 80 includes analyzer 81, synthesizer 82,presentation target selector 83, and presenter 84, as functionalconstituent elements provided by executing the application software.

Analyzer 81 calculates the existence or non-existence (if any, the sizeand the direction) of an influence that each portion of the image(hereinafter, referred to as the input image) as the object detectionprocess target had on each of the object detection results (that is,each of the detection frames) included in the object detection resultsets outputted by the object detection models. Such an influence iscalculated using various methods. The influence can be calculated using,for example, a method disclosed in the following literature. Analyzer 81is an example of the prediction result obtainer and the input datainfluence obtainer, in the present embodiment. Reference: DenisGudovskiy, Alec Hodgkinson, Takuya Yamaguchi, Yasunori Ishii, and SotaroTsukizawa; “Explain to Fix: A Framework to Interpret and Correct DNNObject Detector Predictions”; arXiv:1811.08011v1; Nov. 19, 2018.

FIG. 9A is a schematic diagram for describing the calculation of theinfluence by analyzer 81. In FIG. 9A, a broken line frame in the inputimage represents a detection frame corresponding to an object detectionresult. An illustration to the left of the input image is an extractionof a region including the detection frame, and squares therein arepixels (part of which is omitted) constituting the input image. Anumerical value in each square is a value (hereinafter, also referred toas an analysis value) representing an influence that a value of eachpixel had on the object detection result, which is calculated byanalyzer 81. Note that, for convenience of description, although theanalysis value is put in each pixel in FIG. 9A, such data is notactually added to the value of each pixel. Moreover, althoughdescription is given above assuming that the analysis value iscalculated for each pixel, the present disclosure is not limitedthereto. Pixels may be grouped into a super pixel, and then the analysisvalue may be calculated for each super pixel.

The range of pixels for which the analysis value is calculated differsdepending on the design value of the object detection model and the sizeof the detection frame. In the example illustrated in FIG. 9A, theanalysis value is calculated for a range that is wider by two pixels inthe top-bottom direction and four pixels in the left-right directionthan the pixels overlapping with the detection frame. Theabove-mentioned analysis frame corresponds to the outline of: a set ofall the pixels for which the influence is calculated; or a set of thepixels for which the size of the calculated influence falls within aspecific range.

FIG. 9B is a diagram illustrating an example of the data of the objectdetection result set in which the analysis values calculated by analyzer81 are added to the object detection results. In this example, data ofthe analysis values calculated by analyzer 81 as illustrated in FIG. 9Ais inserted in the data of the object detection result set that isoutputted by Model A and is also illustrated in FIG. 3B. What is in thesecond row in FIG. 9B is data indicating the position and the size of adetection frame as well as the class of a detected subject on which thisdetection frame is put. What is in the third row in FIG. 9B is the data(the middle of which is omitted) of the analysis values calculated forthis detection frame by analyzer 81. What follows in the fourth andsubsequent rows is the position and the size of another detection frameincluded in the data of this object detection result set and the classof a detected subject as well as the data of the analysis valuescalculated for this detection frame.

Based on the analysis values calculated by analyzer 81, synthesizer 82synthesizes an image (that is, a display image) by superposing ananalysis frame on the input image, and outputs the display image. In thepresent disclosure, this analysis frame is represented by such apatterned region as illustrated in each of FIG. 1 and FIG. 7A to FIG.7D. In the case where detection frames exist on one input image, onedisplay image is synthesized for each detection frame. Each displayimage thus synthesized shows one input image together with the influencethat the input image had on the object detection results executed onthis input image, the influence corresponding to the analysis resultcalculated by analyzer 81.

In response to an input from operator 2050 included in UI screen 10displayed on display device 60 such as a monitor, presentation targetselector 83 selects display images to be displayed in result display 40via presenter 84. Operator 2050 is a component on UI screen 10 thatreceives an operation of selecting, by the user, a group of the objectdetection results for which the analysis results are to be included inthe display images to be displayed in result display 40. This group isdefined by the attributes of the object detection results, for example,the class of the detected subject and the result type. Operator 2050 inthe present embodiment includes input data selector 20A, model selector20B, display result selector 20C, display data switcher 50A, analysisframe collective switcher 50B, and analysis frame individual switcher50C. The input from operator 2050 includes: information on the inputimages and the models selected by the user; and information on the groupof the object detection results, that is, information on the class ofthe detected subject or the result type. In accordance with these piecesof information, presentation target selector 83 selects images to bedisplayed in result display 40, from the display images outputted bysynthesizer 82. Moreover, presentation target selector 83 rearranges theselected display images in each object detection result set anddetermines combinations thereof. Further, presentation target selector83 inserts substitution information in an object detection result setincluding no detection result for a detected subject common to othermodels, depending on settings or specifications. Presentation targetselector 83 is an example of the determiner, in the present embodiment.

Presenter 84 outputs, to display device 60, the display images includedin the same combination among the combinations determined bypresentation target selector 83 such that these display images aredisplayed side by side in result display 40. Otherwise, presenter 84outputs the substitution data such that the substitution data isdisplayed in result display 40 side by side with the display images,depending on settings or specifications. Presenter 84 and result display40 of display device 60 are an example of the influence presenter, inthe present embodiment. Moreover, display device 60 is an example of thepresentation device, in the present embodiment.

Procedures of the information processing method according to the presentembodiment performed by computer 100 having such a configuration asdescribed above are illustrated in a flow chart of FIG. 10. Note thatdescription is given below assuming that processing by theabove-mentioned functional constituent elements is processing bycomputer 100.

Computer 100 obtains object detection result sets that are respectivelyoutputted by executing object detection on a common input image bydifferent object detection models (Step S11). Each of the objectdetection result sets includes output values concerning individualdetection frames corresponding to object detection results (see FIG.3B). Note that what actually obtains data of the input image andexecutes the object detection on the data by the object detection modelsmay be computer 100 itself and may be a device other than computer 100.For example, an information processing terminal such as a smartphone oran information processing device included in a device such as a cameramay execute the object detection on an image taken by an imaging elementincluded in each device.

Next, for each of the detection frames corresponding to the objectdetection results included in each object detection result set, computer100 calculates an influence that each portion of the input image had onthe calculation of the output value concerning the detection frame (StepS12). The calculation of the influence specifically means calculating ananalysis value indicating any or both of the size (including theexistence or non-existence) and the direction of this influence.

Next, computer 100 synthesizes display images by superposing theinfluences calculated in Step S12 on the input image (Step S13).Specifically, based on the calculated analysis values, the displayimages are synthesized by superposing analysis frames on the inputimage.

Next, computer 100 determines combinations of the detection results(detection frames) included in the object detection result setsrespectively outputted by the different object detection models (StepS14). The detection results that are determined here to be included inthe same combination are those that are determined to be detectionresults for a common detected subject or detection results whosepossibility of being detection results for a common detected subject ishigher than a predetermined reference, based on overlap of or apositional relationship between the detection frames. Moreover, thecourse of determining the combinations may include insertingsubstitution information in the object detection result sets, dependingon settings or specifications.

Lastly, computer 100 performs presentation of the display imagesselected in accordance with settings of a group of the object detectionresults that is defined by the attributes such as the class and theresult type (Step S15). This selection is performed in units of thecombinations determined in Step S14, and hence the display imagesincluding analysis results for the detection results of the commondetected subject (or the detected subject whose possibility of beingcommon is high) are presented side by side.

Note that the above-mentioned procedures are given as an example, andthe procedures of the information processing method of the presentembodiment are not limited thereto. For example, the synthesizing of thedisplay images (S13) or the obtaining of the influence (S12) and thesynthesizing of the display images (S13) may be executed after thedetermining the combinations (S14) and before the display images areactually presented. In this case, the input image to be included in thedisplay images to be displayed in result display 40 is first selectedbased on the object detection results. Then, the obtaining of theinfluence and the synthesizing of the display images may be performedwhile targeting at only the object detection results for the selectedinput image. Moreover, a user's operation concerning group settings onUI screen 10 may be received at arbitrary timing before Step S15, anddisplay targets selected in accordance with the settings may becomeprocessing targets of unexecuted procedures. Then, in the case where theuser's operation concerning the group settings is received again in orafter Step S15, the display content in result display 40 is switched tothe display images based on the object detection results selected inaccordance with the latest group settings. Moreover, without executionof the determining of the combinations (Step S14), the display imagesmay be displayed once. After that, for example, when a request operationof rearrangement from the user is received, Step S14 may be executed,and the display content in result display 40 may be updated inaccordance with the result thereof. Moreover, the display content inwhich the result of Step S14 is reflected and the display content inwhich it is not reflected may be reversibly switchable.

Moreover, although the expression of the combinations of the objectdetection results (detection frames) is used in the above description,what this expression means in the present disclosure also includescombinations of the input images or the display images determined basedon the overlap of or the positional relationship between the detectionframes.

[Variations and Other Supplementary Notes]

The information processing methods according to one or more aspects ofthe present disclosure are not limited to the description of the aboveembodiment. Various modifications to the above embodiment that areconceivable to those skilled in the art may be included within one ormore aspects of the present disclosure, so long as they do not departfrom the essence of the present disclosure. The following gives examplesof such modifications and other supplementary notes on the descriptionof the embodiment.

(1) In the above-mentioned embodiment, the calculation of the influenceis executed by computer 100, but is not limited thereto. The calculationof the influence may be executed by a device that is other than computer100 and includes an information processing device, and computer 100 maydirectly or indirectly receive and obtain an input of the influenceoutputted by this device.

Moreover, in the above-mentioned embodiment, only a region or a frame(analysis frame) along the outline thereof is given as an example of anexpression aspect of the influence, but the expression aspect thereof isnot limited thereto. For example, the influence may be expressed by afigure such as a point or a cross placed in the center, the geometriccenter, or the like of the region.

(2) In the description of the above-mentioned embodiment and thedrawings referred to in the description thereof, the display images inwhich the object detection results included in the same combination andthe influences respectively obtained for these object detection resultsare superposed on each other are presented side by side in one row inresult display 40, but how to present the display images is not limitedthereto. For example, the display images may be presented one above theother in one column, and may be presented while being arranged in amatrix formed from rows and columns, such as a cross-in-square shape.Moreover, for example, the detection frames corresponding to the objectdetection results by each of the models or the analysis frames obtainedfor the detection frames may be collectively superposed on one inputimage. The analysis frames in this case are not such patterned regionsas illustrated in FIG. 7A to FIG. 7D, and may be each represented byonly an outline or may be each represented by a region that is coloredwith a higher degree of transparency.

(3) In the description of the above-mentioned embodiment and thedrawings referred to in the description thereof, given is the examplewhere the display images in which the influences obtained for the objectdetection results included in the determined same combination aresuperposed on the input image are presented side by side, but what ispresented in result display 40 is not limited to such display images.For example, the input image on which the detection frames correspondingto the object detection results are superposed instead of or in additionto the influences may be presented in result display 40. Moreover, theinformation to be superposed on the input image may be switchable inresponse to the user's operation.

(4) In the description of the above-mentioned embodiment and thedrawings referred to in the description thereof, for convenience ofdescription, what is displayed at a time in result display 40 on UIscreen 10 is only the object detection results included in onecombination and the influences obtained for these object detectionresults, but is not limited thereto. If the display images based on theobject detection results included in the same combination are gathered,arranged, and further surrounded by a frame and thus can be easilyunderstood by the user, the display images corresponding to combinationsmay be presented at a time in result display 40.

(5) In the description of the above-mentioned embodiment, thecombinations are determined by rearranging and placing, in the sameorder, the contents of the data of the object detection result sets, buthow to achieve the combinations is not limited thereto. For example,instead of rearranging the contents of the data of the object detectionresult sets, an identifier indicating the combination to which eachobject detection result belongs may be added, or a table in whichinformation on the combinations is held may be generated or updatedseparately from the object detection result sets. Alternatively,calculation may be performed each time in response to the user'soperation, and the combinations for determining the display images to bepresented in result display 40 may be determined.

(6) The sorting depending on the classes of the detected subjects, whichis described with reference to FIG. 3A and FIG. 4A, may be executed notonly to compare the prediction results among the models but also toorganize the prediction results by one model.

On the other hand, even if the prediction results by the models or theanalysis results thereof are display targets, the sorting depending onthe classes is not essential. That is, without the sorting depending onthe classes, the determining of the combinations based on the overlap ofor the positional relationship between the detection frames may beperformed.

(7) In the description of the above-mentioned embodiment using theexamples illustrated in FIG. 3A to FIG. 6, the determining of thecombinations of the object detection results is performed based on theoverlap of or the positional relationship between the detection frames,but the method of determining the combinations is not limited thereto.For example, in the case where the influences, that is, the analysisframes have already been calculated, the combinations may be determinedbased on the degree of similarity among the influences. The degree ofsimilarity among the influences here can be determined, for example, bycomparing the magnitudes of vectors or based on the degree of similarity(for example, a cosine distance) between these vectors, the vectors eachhaving, as elements, the (horizontal and vertical) size of a pixelregion for which analyzer 81 calculates the analysis value for each ofthe object detection results respectively outputted by two objectdetection models.

(8) Moreover, in addition to the overlap or the positions of thedetection frames, the combinations of the object detection results maybe determined further based on the degree of similarity betweenlikelihoods of the detection results. For example, higher scores may begiven to combinations with larger overlap of the detection frames andcombinations with a higher degree of similarity between the likelihoodsof the detection results, and the combinations may be determined basedon the result of such scoring.

(9) A portion or all of the functional constituent elements included inthe above-described information processing systems may be configuredfrom one system large-scale integration (LSI) circuit. A system LSIcircuit is a super-multifunction LSI circuit manufactured with aplurality of components integrated on a single chip, and is specificallya computer system including a microprocessor, read-only memory (ROM),and random access memory (RAM), for example. A computer program isstored in the ROM. The system LSI circuit achieves its function as aresult of the microprocessor operating according to the computerprogram.

Note that although a system LSI circuit is described here, it may alsobe referred to as an integrated circuit (IC), an LSI circuit, a superLSI circuit, or an ultra LSI circuit, depending on the degree ofintegration. Moreover, the circuit integration technique is not limitedto LSI, and may be realized by a dedicated circuit or a general purposeprocessor. After manufacturing of the LSI circuit, a field programmablegate array (FPGA) or a reconfigurable processor which is reconfigurablein connection or settings of circuit cells inside the LSI circuit may beused.

Further, when development of a semiconductor technology or anotherderivative technology provides a circuit integration technology whichreplaces LSI, functional blocks may be integrated by using thistechnology. Application of biotechnology, for example, is a possibility.

(10) An aspect of the present disclosure is not limited to theinformation processing method described with reference to the flow chartillustrated in FIG. 10, and may be an information processing system thatincludes computer 100 and a program executed by computer 100. Inaddition, an aspect of the present disclosure may be a non-transitorycomputer-readable recording medium having the computer program recordedthereon.

INDUSTRIAL APPLICABILITY

An information processing method and the like according to the presentdisclosure can be utilized for a user interface used to compare resultsof a prediction process by a computer.

1. An information processing method performed by a computer, theinformation processing method comprising: obtaining prediction resultsthat are results of prediction performed by predictors on same inputdata; obtaining, for each of the prediction results, an influence thatthe input data had on the prediction result; determining, based on theprediction results, one or more combinations of the prediction results;and presenting, side by side or by superposition, using a presentationdevice, influences obtained for prediction results that are included inthe prediction results and are in a same combination among the one ormore combinations.
 2. The information processing method according toclaim 1, wherein each of the predictors is an object detector, each ofthe prediction results is an object detection result set that includesobject detection results, the influence obtained for each of theprediction results is an influence that the input data had on each ofthe object detection results included in the object detection resultset, each of the one or more combinations includes object detectionresults included in different object detection result sets among theobject detection result sets, and the influences presented side by sideor by superposition are influences obtained for the object detectionresults included in the same combination.
 3. The information processingmethod according to claim 2, wherein each of the object detectionresults includes a class that is based on an object detected, and ineach combination among the one or more combinations, the class includedin each of the object detection results included in the combination is aclass that is common to other object detection results included in thecombination.
 4. The information processing method according to claim 2,wherein each of the object detection results includes a detection frame,and the one or more combinations are determined based on overlap of or apositional relationship between detection frames included in the objectdetection results included in different object detection result sets. 5.The information processing method according to claim 4, wherein each ofthe object detection results includes a detection likelihood, and theone or more combinations are determined further based on a degree ofsimilarity between detection likelihoods included in the objectdetection results included in different object detection result sets. 6.The information processing method according to claim 2, wherein thepresenting of the influences includes presenting, side by side,substitution data and the influences obtained for the object detectionresults included in the same combination, when the same combination doesnot include the object detection results included in an object detectionresult set of an object detector among the object detectors.
 7. Theinformation processing method according to claim 2, further comprising:presenting, side by side, substitution data and an influence obtainedfor an isolated object detection result when the object detectionresults include the isolated object detection result, the isolatedobject detection result being an object detection result not included inany of the one or more combinations.
 8. The information processingmethod according to claim 2, wherein the presenting of the influencesincludes presenting: an influence obtained for a reference objectdetection result that is one of the object detection results included ina reference object detection result set that is one of the objectdetection result sets; and influences obtained for the object detectionresults included in a combination among the one or more combinationsthat includes the reference object detection result.
 9. The informationprocessing method according to claim 8, further comprising: receiving anoperation of selecting the reference object detection result set; andswitching the reference object detection result set to the objectdetection result set selected by the operation.
 10. The informationprocessing method according to claim 1, further comprising: receiving anoperation of selecting the input data; and switching the input data tothe input data selected by the operation, wherein the presenting of theinfluences includes presenting influences obtained for the predictionresults of prediction performed by the predictors on the input dataselected.
 11. The information processing method according to claim 2,further comprising: receiving an operation of selecting a group of theobject detection results; and switching the object detection resultscorresponding to the influences presented or the object detectionresults presented, to object detection results of the group selected bythe operation.
 12. The information processing method according to claim1, wherein the presenting of the influences further includes presenting,for each of the influences presented, information indicating a predictoramong the predictors that has output the prediction result correspondingto the influence.
 13. An information processing method performed by acomputer, the information processing method comprising: obtaining objectdetection result sets each including object detection results that areresults of detection performed by object detectors on same input data;determining, based on the object detection results included in theobject detection result sets, one or more combinations of the objectdetection results included in the object detection result sets, the oneor more combinations each including one object detection result of eachof the object detection result sets; and presenting, side by side or bysuperposition, using a presentation device, object detection resultsthat are included in the object detection results in the objectdetection result sets and are in a same combination among the one ormore combinations.
 14. An information processing system comprising: aprediction result obtainer that obtains prediction results that areresults of prediction performed by predictors on same input data; aninput data influence obtainer that obtains, for each of the predictionresults, an influence that the input data had on the prediction result;a determiner that determines, based on the prediction results, one ormore combinations of the prediction results; and an influence presenterthat presents, side by side or by superposition, using a presentationdevice, influences obtained for prediction results that are included inthe prediction results and are in a same combination among the one ormore combinations.
 15. A non-transitory computer-readable recordingmedium for use in a computer which includes an information processingdevice, the recording medium having a program recorded thereon forcausing a processor included in the information processing device to:obtain prediction results that are results of prediction performed bypredictors on same input data; obtain, for each of the predictionresults, an influence that the input data had on the prediction result;determine, based on the prediction results, one or more combinations ofthe prediction results; and present, side by side or by superposition,using a presentation device, influences obtained for prediction resultsthat are included in the prediction results and are in a samecombination among the one or more combinations.